GROUP ACTION ON INSTANTON BUNDLES 

OVER p3 t\ 

LAURA COSTA Q GIORGIO OTTAVIANI Q 
March, 2001 

Abstract: Denote by MI{k) the moduh space of fc-instanton bundles E of 
rank 2 on P'^ = P{V) and by Zk{E) the scheme of fc-jumping hnes. We prove 
that [E] e MI{k) is not stable for the action of SL{V) if Zk{E) ^ 0. Moreover 
dim Sym{E) > 1 if lengthZk{E) > 2. We prove also that E is special if and only if 
Zk{E) is a smooth conic. The action of SL[V) on the moduli of special instanton 
bundles is studied in detail. 

1 Introduction 

A fc-instanton bundle on = F{V) is a stable 2-bundle E such that ci = 0, 
C2 = k and H^{E{-2)) = (see [|Har78|| and ||BH78|| ). The instanton bundles 



which are trivial on the fibers of the twistor map P^ 5^ correspond to self dual 
Yang Mills S'p(l)-connections on S'^, according to the ADHM-correspondence. The 
moduli space of fc-instanton bundles on P^ will be denoted by M/p3(fc). 

Our aim is to study the natural action of GL{V) over M/p3(fc). In this sense this 
paper can be seen as the natural continuation of the study begun in | |A099| trying to 



answer a general problem raised by Simpson who asked about the stable points of the 
GL(V^)-action on the moduli space of bundles over V{V). In [ICOOOII we proved that 



M/p3(fc) is an affine variety and this fact simplifies a lot the geometry of the group 
action. An easy consequence is that all points of M/p3(fc) are semistable according 
to Mumford's geometric invariant theory. For A; > 3 the generic [E] G M/p3(/c) has 
trivial stabilizer Sym{E) and hence, for k > 3, among the nonstable points [E] are 
all the points [E] such that Sym{E) has positive dimension. In this work, we will 
focus our attention on the connected component Sym^{E) containing the identity. 

A classical tool for the study of vector bundles is the scheme of jumping lines 
(see ||OSS80|| ). The restriction of a fc-instanton bundle E on a line L is isomorphic to 
Ol{—p)Q)Ol{p) for some p depending on L and satisfying the inequahties < p < k. 
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It is known that p = is the general value and if p > 1 then L is called a jumping 



line of order p. A useful criterion that we prove is the following (see Theorem [4.4| , 
Example |4.6| ) : 

Theorem A: Let E be a k-instanton bundle. If E has a jumping line of maximal 
order k, then [E] is not stable and the converse is not true. 

Moreover, we have found an example of a fc-instanton bundle E with Zk{E) ^ 
and Sym^{E) = (see Example 



In |ST90| was observed that the action of SL{2) ~ SL{U) over V ^ U ® U is 



useful to get the cohomology and other geometrical properties of special instanton 
bundles. Unfortunately special instanton bundles are not invariant for such action. 
Instead, they are invariant for the action of SL{2) ~ SL{U) over ~ f/ 
where SL{U) acts trivially over C^. Our starting point was the remark that the 
correct action to be considered is SL{2) x SL{2) ~ SL{U) x SL{U') ~ SpiniV) 
acting over ~ f/ (g) f/' for a suitable isomorphism (see Remark |4.2| ), hence for a 
suitable symmetric form on V . In fact the symmetry group of the kernel bundle 
considered in ||ST90|| is SL{U) x SL{U') and for any special instanton bundle E we 
get that Sym{E) lies always in such a group (we believe that this fact is true for any 
instanton bundle). This fact conduces us to introduce the definition of fine action 
(see Definition |47^ ). In particular a subgroup G C GLiV) has a fine action if there 
are 2-dimensional vector spaces U and U' such that G C GL{U) x GL{U') and G 
acts according to fixed isomorphism ~ f/ ®U' . 

In Section 5 (see Theorem |5.1| and Theorem |5.3| ) we will prove 

Theorem B: Let E be a k-instanton bundle. If length Zk{E) > 2 then Sym^{E) 
contains G of positive dimension having a fine action. 



In addition, by means of Example we will see that the converse of Theorem 
B is not true. On the other hand, we will see 

Theorem C: The following are equivalent 

(i) Zk{E) is a conic. 

(ii) E is a special k-instanton bundle. 

(Hi) SymP{E) ^ SL{U') having a fine action. 
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The equivalence between (i) and (ii) is proved in Corollary |2.9| , (ii) implies (Hi) 
is proved in (j^) meanwile {Hi) implies (ii) is proved in Proposition [4.13| . 

After this result, we are led to conjecture that for any instanton bundle E, 
SyimP{E) has a fine action. 

In Section 4 we study the group action over the subvariety MI^o,{k) of special 
fc-instanton bundles (see Definition |2.4| ) and our main results are Corollary [4.12| and 
Theorem [4.14| . More precisely, it is known that every special fc-instanton bundle is 
defined by {k + 1) lines through the classical Serre correspondence and that there 
is a pencil of lines whose branch locus can be studied with the classical machinery 
of binary forms and which is a GL(V^)-invariant. In Theorem [4.14| we show that in 



general there are exactly ^^(fc) Gi^(^)-orbits of special fc-instanton bundles with 



the same branch locus. We mainly apply results of Trautmann ( [P?ra88|| ) that we 
report in a form suitable for our purposes. Although Trautmann did not apply his 
results to instanton bundles, probably because the group action approach on moduli 
has only nowadays a stronger importance, some of these applications were certainly 
known to him. 

We want to point out that we have chosen to present the actions in matrix form, 
due to the fact that this is suitable in view of computer implementing ( [|Anc96|| ). 



2 Generalities about instantons over 

The goal of this section is to prove some general results about fc-instanton bundles 
on that will be needed later. To this end, we start fixing some notation and 
recalling some known facts (See for instance ||BT87|| ). 



Notation 2.1. 0{d) = 0-p3{d) denotes the invertible sheaf of degree ci on P^ = 
P(V^). For any coherent sheaf E on P^ we denote E{d) = E® Cp3(rf) and by K'E{d) 
the dimension of H\^^, E{d)) = H'E{d). 

Definition 2.2. A k-instanton bundle on P^ is an algebraic vector bundle E of rank 
2 with Chern classes Ci{E) = and C2{E) = k, which is stable (i.e. h^E = 0) and 
satisfies the vanishing h^E{—2) = 0. We will denote by MIps{k) the moduli space 
of k-instanton bundles on P^. 



Let E he a fc-instanton bundle. It is well known (see [ BH78 |) that there is a 
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vector space I of dimension k and a symplectic vector space (VF, J) of dimension 
2k + 2 such that E is the cohomology of a symplectic monad 



(1) I* ®0{-l)^W ®0^1®0{l). 

That is, E ~ KerA/ImA^, where the transpose A* is computed with respect to 
the symplectic form J. We have isomorphisms / ~ H^{E{—1)), W ~ H^{E ® Q^). 

The vector space ® / ® = Hom(W ®0,I® 0{\)) contains the subvariety 
Q given by morphisms A such that the sequence (0) is a complex, that is, such that 
AJA^ = 0. In that case, we shall say that A represents E. GL{I) x Sp{W) acts on 
Q by {g, s) ■ A = gAs. Let Q° be the open subvariety oiQCW®l0V which 
consists of morphisms A E Q that are surjective. By [ [BH78| ] (see also ||CO00|| ) all 



points of Q° are stable for the action of GL{I) x Sp{W) and M/p3(/c) is isomorphic 
to the geometric quotient Q^/GL{I) x Sp{W) according to Mumford's GIT. In 
particular any morphism between fc-instanton bundles lifts to a morphism between 
the corresponding monads. 

Definition 2.3. Let E be a k-instanton bundle on P^. A hyperplane if C zs an 
unstable hyperplane of E ifh^E\H 7^ 0. The set of unstable hyperplanes of E has a 
natural structure of scheme and it will be denoted by W{E) C p^*. 



The fact that, for any fc-instanton bundle E on P^, h^E{l) < 2 ( ||BT87]| ) allows 
to pose the following definition 

Definition 2.4. A k-instanton bundle E on P^ is called special if h^E{l) = 2. 

We denote by Ml^^ik) the moduli space of special /c-instanton bundles. By 
its definition, it is constructed as a closed subscheme of the affine variety MJp3(fc) 
( ||CO00|| ). In particular, MIt,^{k) is also affine and it has dimension 2/c + 9 (see e.g. 



Proposition |2]T|). Notice that any E e M/p3(2) is special ( ||BT87|| ). 



For any A;-instanton bundle E we have dimW{E) < 2. By a theorem of Coanda 
( ||Coa92|| ) a /c-instanton bundle E is special if and only if dimW{E) = 2 and in this 
case W{E) is a smooth quadric. 

Now we are going to give a similar result for the scheme of /c-jumping lines. 

Definition 2.5. Let E be a k-instanton bundle on P^ and 1 < p G Z. A line I on 
is called a p-jumping line of E if E\i = Oi{—p) © Oi{p) and we denote by 

Z,{E) = { leG{F\F') I Eu = Oi{-z)®Oi{i) 3<i } 
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being G{P^,P^) the Grassmaniann of lines on P^. 

Zj{E) has a natural structure of closed subscheme of G{P^,F^). From the definition 
it follows that for any A;-instanton bundle E on there is the following filtration 

G(p\p3) = Zo{E) D Zi{E) D Z2{E).... 



Moreover, by |^ki97|| ; Proposition 13 and [ blao97 |, Zk+i{E) = 0. Hence a line in 
Zk{E) is called a maximal order jumping line. 

Remark 2.6. Zk{E) is obtained by cutting the Pliicker quadric (^4 ^ p^ with 
a linear space. In particular deg Zk{E) < 2. Let (xq, . . . ,X3) be homogeneous 
coordinates on F{V) and let A = Yl^=o ^W^I^Vhea matrix representing 
E. Then it is well known that Zk{E) is obtained by cutting the Pliicker quadric 
- P02P13 + P03P12 = with the linear space Y.i<jPij^iJ ^] = 0- 

In the following result we relate the maximal order jumping lines of E with 
unstable hyperplanes of E. 

Lemma 2.7. Let E be a k-instanton bundle on P^ and I G Zk{E). Then any H D I 
is an unstable hyperplane of E, i.e., if G W{E). 

Proof. Since H D I we have the following exact sequence 

^ E{-1)\H ^ E\H ^ E\, ^ 0. 

Assume that H is not an unstable plane. Taking cohomology to the above ex- 
act sequence we get the injection H^E\i ^ H^E{—1)\h. Since by Serre's duality 
h'^E{-l)iH = h'^E{-2)\H = 0, this gives us 

k + l = h^E\i < h^E{-l)\H = -x{E{-1)\h) = k 

which is a contradiction. Hence H is an unstable hyperplane of E. □ 

Remark 2.8. In particular, since for any fc-instanton bundle E on P^ we have 
h^E\H < 1 ( [|K099|| ; Theorem 1.2), we get immediately from the above proof that 
Zj{E) = for any j > A; + 1. 

Corollary 2.9. Let E be a k-instanton bundle on P^. We have dimZk{E) < 1. E 
is special if and only if dimZk{E) = 1 and in this case Zk{E) is a smooth conic in 
G(p\p3). 
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Proof. It follows from Lemma |2.7] that if dimZk{E) > 1 then dimW{E) > 2. 
Hence, by [|Coa92|| , E is a special instanton bundle. □ 



Remark 2.10. Let E he a fc-instanton bundle on P^. If has three different k- 
jumping lines then E is a. special fc-instanton bundle. 



We know ( ||Har78|| for A; = 2 and ||B'L87|| in general) that ii E e Ml^^ik) then 
Zk{E) is a smooth conic on G{F^,P^) and that W{E) is a smooth quadric. In 
addition, Z^i^E) corresponds to one of the two rulings of W{E) and the generic 
s G H^E{1) vanishes on k disjoint lines of the other ruling. Moreover, it is well- 
known that each E G Ml^^ik) can be identified with a g\j^^ without base points on 
Zk{E) ^ and that we can choose this line as the line of all hyperplanes containing 
the line {xq = a;i = 0}. In fact, the dual of a smooth quadric is isomorphic to 
the quadric itself, and we can consider C p^ as the dual of W{E) and then the 
tangent hyperplanes to Qe belong to W{E). The gl_^_-^ is constructed by cutting with 
the zero locus of s G H^E{1) on a fixed line of the ruling in W{E) corresponding 
to Zk{E). We will define the branch locus of this g\j^i as the branch locus of the 
bundle. It is defined up to S'L(2)-action. 



Notation 2.11. We will denote by the open subset in G(P^,P^) which consists 
of 

Qk+i without fixed points and we will denote by W C Gr(P^,P^) the open subset 
of planes which cut the Pliicker quadric in a smooth conic. 

The above discussion shows the following 

Proposition 2.12. f ^HM^J , f^W\] ;Coroll4.7} The morphism ni: Ml^-^ik) ^ W 
which takes E to the plane spanned by the conic Zk{E) is a fibration such that all 
the fibers are isomorphic to . 



Let Y C p(S'^V) ~ P^ be the open subvariety of smooth quadrics in P(V^). There 
is another fibration 7^2 '■ MI^3{k) Y which takes E to the quadric W{E). All the 
fibers of tt2 are isomorphic to the disjoint union of two copies of G^ and we have a 
commutative diagram 

MI^,ik) ^ Y 

II ij 
MlUk) ^ W 
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where j is a 2 : 1 finite map. In addition, for any Q G W the fiber j ^{Q) is given 
by the two ruhngs of Q 



1,1 X pi. 



In ||A098|| ; Proposition 1.1, special /c-instanton bundles were characterized in 
terms of their monads. However that description, as the one given in | |B'1'87| |, is not 
useful for our purposes because the monads considered there were not symplectic and 
the techniques that we develop in Section 4 do not apply. So we have to introduce 
an alternative description. With this aim we introduce some notations 

Definition 2.13. A {k + 1) x [k + 1)-Hankel matrix H = (aij) = (aj+j_2) is one 
having equal elements along each diagonal line parallel to the secondary diagonal, 
i.e 



H 



( «o 

Ci2 



0L2 



Ol2 



ttfc+l ttfc+2 



ftfc-l Oik 



Oi2k-l Oi2k J 



Let X = [a;^ -kx^^^y, Qx'^-'^y^, ... , (-1) V] and H he a {k + 1) x {k + 1)- 
Hankel matrix. Following [[TraSSH ; 2.4, we denote by fn the form of degree 2k defined 
by X -H-XK 

Definition 2.14. Denote by A the invariant for fu defined by the determinant 



Oi2 



ak 
ttfc+i 



Ofc Ofc+i • • • a2k 
Let (xo, . . . , xs) be homogeneous coordinates on P(y). We set 



4(xo,xi 



/ Xq Xi 
\ Xq Xi J 



and /fc(xo,xi) 



( Xi \ 

Xo Xi 

xq Xi 
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The kx [k + l)-matrix and the {k + 2) x [k + l)-matrix are both well known 
in the theory of vector bundles. In fact, the matrix defines a surjective morphism 
of vector bundles on = F[U) given by the natural multiplication S'^U ® O 
S^~^U ® 0{1), whose kernel is isomorphic to 0{—k). More interesting is the fact 
that the converse holds. In fact, it is well known (see e.g. ||A099|| ) the following 



Proposition 2.15. Every surjective morphism of vector bundles on 

is represented, in a suitable system of coordinates {xo,Xi), by the matrix Ik- 

Proof. The proposition is equivalent to the fact that there is a unique SL{2) x 
SL{k) X SL{k + l)-orbit for nondegenerate matrices in P(C^ (g) C^^ ® c'''+^). □ 

Notation 2.16. Given H a nondegenerate {k + 1) x {k + 1)-Hankel matrix, we 
denote 

(2) A:=[Ik{xo,xi)\Ik{x2,X3)-H]. 



Lemma 2.17. Let J be the standard skew- symmetric nondegenerate matrix and let 
A be as in The following holds 

(i) AJA* = 0. 

(a) Let E be a k-instanton bundle on defined by A. Then h^E{l) = 2 and E is 
special. 

(Hi) Conversely, every special k-instanton bundle is the cohomology bundle of a 
monad (W for a suitable system of coordinates. 



Proof, (i) is a straightforward verification. Let 



1 \ 



and let 



A 



Ik{Xo, Xi) ■ Tfc+i ■ H ^\Ik{x2, X3) ■ 



8 



Then AJA^ = 0. Since A has {k + 2) rows it follows h^E{l) = 2 which proves 
(ii). When H moves among the nondegenerate Hankel matrices, the moduli space 
of bundles given by is isomorphic to P(5'^'^f/) \ {A = 0} (see Corollary p.2| ). In 
Theorem ^]5| we quote a result of Trautmann which shows that F{S'^^U) \ {A = 0} 
is isomorphic to G^. Since this is exactly a fiber of the map tti of Proposition |2.12| , 
we obtain [in). □ 



Remark 2.18. With the notations of the above proof set 



H 



(^2k ) 



Then, E ■ Tk+2 ■ A = A. 



We will end with the following result which is a straightforward verification and 
we left it to the reader. 

Lemma 2.19. Let E he a special k-instanton bundle defined by the monad 
Then W{E) is given by all the hyperplanes which are tangent to the quadric XqX^ — 
X1X2 and Zk{E) is the conic obtained by cutting the Pliicker quadric with the plane 
{po2 = Pi3 = Po3 + P12 = 0} . □ 



3 Useful facts about /SL(2)-actions 

For any g G 5l{U), we denote by s^g G sl{S^U) the image of g through the Lie 
algebra representation of sl{U) given by the fc-symmetric power. Notice that sl{U) 
acts on S'^{S^U) by 

sl{U) X {S^S'^U)) (S^S'^U)) 

{g,A) ^ {s'^gY ■ A + A - s'^g. 



Lemma 3.1. Let H be a [k + 1) ^ {k + 1)-Hankel matrix determined by a = 
{aQ, a2k) o,nd consider g G sl{U). Then the following holds 

(i) H' = [s^gY ■ H + H ■ s'^g is a Hankel matrix. 
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(a) If a' = («[), ...,a2k) are the coefficients of H' then {s'^'^gY ■ (a)* = 

(in) The natural action of 5l{U) on forms of degree 2k takes fn to f^'- 

Proof. It follows from direct computation. □ 

Corollary 3.2. The [k + l) x {k + 1)-Hankel matrices forms an invariant subspace 
of {S'^{S''U)) isomorphic to S'^^U . 

Proof. It is an immediate consequence of Lemma |3.1| . □ 

There is an analog description for the Lie group SL{U). If (7 G SL{U), we denote 
by S'^g G SL{S'^U) the image of g through the group representation of SL{U) given 
by the /c-symmetric power. 

As a consequence of the fact that S''^'(exp(7) = exp(s^(7), from Lemma RTTI we 
obtain 

Lemma 3.3. Let H he a (k + 1) x {k + 1)-Hankel matrix determined by a = 
(cto, <y2k) and consider g G SL{U). Then the following holds 

(i) H" = {S'^gY ■ H ■ S^g is a Hankel matrix. 

(a) If a" = (a'Q, ...,a2fc) are the coefficients of H" then {S'^^gY ■ (a)* = (a")*- 
(Hi) The natural action of SL{U) on forms of degree 2k takes fn to fn"- □ 



The following is a known result whose generalization can be found in | AU91j 



Proposition 3.4. Let g G SL{2) be such that f ' ( ) = ( ) ■ Then 

{S'-'gYhixo^x^) {S'g-'y = h{x'o,x\). 

□ 

Theorem 3.5. ( ^Tra8^] ; Prop. 2.2) The morphism f{S^^U) \ {A = 0} — > C 
G'(p\p(5'=+if/)) defined by 



/ ao oil ■■■ Ofc-i \ 
ai a2 ■ ■ ■ Oik 



a ^ L„ = {(/o, . . . , fu+i) G ^'=+^?7|(/o, . . . , /fc+i)- 

\ afc+2 ■ ■ ■ a2k J 

being a = {ao, . . . , a2k), is a S L{U)-equivariant isomorphism. □ 



0} 
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We have a filtration of S'L(f/)-invariant closed subvarieties ^ ■ ■ ■ ^ H2 ^ 
Hi = F{S'^''{U)) where Hj = {/ G S^''{U)\f has a root of multiphcity > j}. In 
particular, H2k is the rational normal curve of degree 2k and H2 is the discriminant 
hypersurface. The very basic example of Mumford's GIT (already known to Hilbert) 
shows that Hk+i consists of the locus where all the S'L(f/) -invariant vanish ( i.e. not 
semistable points) and the 5'L([/)-quotient map is defined on = P{S'^^ {U))\Hk+i. 
We note that -fffc+i C {A = 0}. It is well known that the orbit of / G is not 
closed in if and only if / G Hk (i.e. not stable points). In fact, the orbits of 
points in Hk all contain in their closure the orbit of x^y'' G Hk, which is the only 
orbit which is two-dimensional (all the other orbits are three-dimensional). This 
particular S'L(f/) -orbit corresponds to a particular S'L(V^)-orbit of special instanton 
bundles, that we will define in a while. 

Now consider the analogous filtration Wk+i C . . . C ^ H^i = P(S'''+^(f/)) 
where Wj = {/ G S^~^^{U)\f has a root of multiplicity > j}. In particular , Wk+i 
is the rational normal curve of degree k + 1 and W2 is the discriminant hypersurface 
of degree 2k. 

It is well known (and easy to check) that a pencil L G Gr{F^,F{S''^^{U)) has a 
fixed point (that is it does not belong to G'^) if and only if L is contained in some 
osculating space to the rational normal curve Wk+i- Moreover, the branch locus of 
L can be identified with Bl G F{S'^^{U)) being Bl isomorphic to L fl W2- It follows 
the basic fact that 

L n Ws+i ^fl) ^ Bl e H, for 1 < s < 
and we also remark that 

L„ n Wk+i ^^^aeHk 



compare it with | Tra88 |;Prop. 2.6). The map G P{S (U)) which takes L to 



Bl extends to a map 

R: Gr{F\p{S''+\U)) FiS^^U)) 

which is finite and equivariant (see |P?ra88|| Pt Remark 2.5, and also our proof of 
Theorem |4.14| ). In particular, a pencil L G G'^ is not stable if and only if LCiWk+i 7^ 



). Notice once more that all the orbits of L such that L fl Wk+i 7^ contain in the 
closure the unique orbit given by the chordal variety to Wk+i- 



''The only correction to be done in the diagram at page 41 of [ TraSS is that the arrow H must 
be dotted. 
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We resume the above discussion in the following 

Corollary 3.6. All points of are semistable for the action of SL{U) and the 
only not stable ones are the pencils with a point of multiplicity (fc + 1). All the not 
stable orbits contain in the closure the unique orbit of pencils with two points of 
multiplicity {k + 1). □ 



4 The action on MI^i{k) and MI^s{k) 

Along this section, we will keep the notations introduced before. Since a morphism 
between /c-instanton bundles lifts to a morphism between the corresponding monads, 
we get that the action of GL(y) over M/p3 {k) lifts to the action of GL{I) x Sp{W) x 
GL{V) over Q^. This means that if E and E' are fc-instanton bundles in the same 
GL{V)-OThit then any two representatives A, A' G Q° are in the same GL{I) x 
Sp{W) X GL(^)-orbit. The (connected components containing the identity of the) 
stabilizers and the stable points of the two actions correspond to each other (this 
technique was used in |[A099|] ). 



As a consequence of the main theorem in ||CO00|| we get 



Theorem 4.1. All points in MIp3{k) are semistable for the GL{V)-action. 

Proof. Since Q° is the complement in Q of an invariant hypersurface (see ||CO00|] ) 
all points in Q° are semistable. In addition, the two GIT-quotient MIp3{k)/GL{V) 
and Q^/GL{I) x Sp{W) x GL{V) are isomorphic. □ 

Remark 4.2. Let U and U' be vector spaces of dimension 2. Fix three isomor- 
phisms y ~ f/ ?7', ^ S^U ® U' and I ^ S'^^^U. Then, there is an induced 
natural muhiplication map A e Hom{W ®0,I® 0{l)) which is GL{U) x GL{U')- 
invariant. Indeed it can be shown that kerA is a rank k + 2 bundle on ^(V) with 
symmetry group in GL(y) isomorphic to GL{U) x GL{U')/C* . In the case U = U', 
this construction was considered in ||ST90||, where kei A was called the kernel bun- 



dle. We could work in SL{V), which is the universal cover of Aut{¥{V)) , and in this 
case the symmetry group of ker A is SpiniV) ~ SL{U) x SL{U'), but we prefer to 
consider the GL groups, which is a not essential variation. 

After this remark, we are led to pose the following definition, which will play an 
important role in the sequel. 
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Definition 4.3. A subgroup G C GL(y) is said to have a fine action over Mlpa^k) 
if there are vector spaces U and U' of dimension 2 such that G C GL{U) x GL{U') 
and there exists a lifted action of G over W ® I ®V which is determined by some 
fixed isomorphisms V ^ U U' , W ^ S'^U ® U' , I ~ S'^^^U. 

Theorem 4.4. Let E be a k-instanton bundle such that Zk{E) ^ 0. Then [E] is a 
not stable point in MIp3{k) for the GL(y)-action. 



Proof. By [ [Ski97 |, E is represented by 

A= [I{xo,Xi)\H{xo,Xi,X2,X3)] 

where H is a k x [k + 1) matrix of hnear forms. Let A ^ C* C SL{U') acting in 
such a way that the splitting W = JJ'^^^ defines two eigenspaces in W with positive 
and negative weights that correspond to the two blocks in which are divided the 
columns of A. Moreover, the positive eigenspace on V = U' Q) U' is generated by 
xo, xi and the negative one by X2, X3. The action on / is trivial. C* has a fine action 
according to Definition ^]3| and we get 



lim A(t) ■ A = [/(xo, Xi)\H{0, 0, X2, X3)] 



which still represents a fc-instanton. □ 
Remark 4.5. By ||Kao97[ we know that if Zk{E) 7^ then [E] is a smooth point 



in M/p3(/c). It should be interesting to know if the unstable points for the GL(y)- 
action over MIm3{k) are smooth points. 



Let us see, by means of an example, that the converse of Theorem ^]4| is no 
longer true. 

Example 4.6. Let 

5X3 X2 

A = I 5xo xi 5x3 X2 

3X3 X2 4xo^ 

It is easy to check that AJA^ = and that the associated morphism is surjective 
(the rank is 3 on every (xq, . . . , X3) G P^) so that A represents a 3-instanton bundle 
E. Moreover Z-^lE) = and C* C Sym{E) having a fine action. The weights are as 
follows: 
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• on J : -2,0,2 



• onW : 5,3,1,-1,-5,-3,-1,1 

• on K : a;o(-3),Xi(-l),X2(l),X3(3) 

which correspond to weights —1 and 1 on f/ and —2 and 2 on U' with the isomor- 
phisms / ~ S'^U, W ~ S^U (g)f/', \/ ~ U ®U' . In particular [E] is a not stable point 
for the GL(V^)-action over MI^3{k) and this shows that the converse of Theorem 
is not true. 



Example 4.7. The generic 3-instanton bundle E with Z3 given by one point has 
Sym^{E) = 0. This can be checked with the help of Macaulay system ( ||BS|| ) by 
using Skiti monad ( [pki971] ) with a generic Hankel matrix. Moreover, it seems likely 
that the generic A;-instanton bundle E with Zk ^ ^ has SyimP{E) = 0. On the other 
hand, there are examples of fc-instanton bundles such that dim Sym{E) = 2. 



Keeping the notations introduced in Section 2, we recall a result from [ |(JUOCI| |. 

Theorem 4.8. There is a Sp{W) x SL{I) x S L[V) -invariant homogeneous polyno- 
mial D over W ® I ®V of degree 2k{k + 1) such that A E Q'^ if and only if A E Q 
andD{A)^0. □ 



Corollary 4.9. Let A E representing a k-instanton bundle E and A: C* ^ 
Sp{W) X SL{I) X SL{V) be a morphism such that its image is contained in Sym{E). 
Then, z/lim^^o A(t) ■ A exists, it belongs to . 

Proof. It follows from the fact that D{X{t) ■ A) is constant with respect to t. □ 

Remark 4.10. After Corollary |4.9| , a classification of all C*-invariant fc-instanton 
bundles should give the classification of unstable points in Ml-ps^k), but we postpone 
this study. 



For special fc-instanton bundles the description of the group action is quite pre- 
cise. By the above geometric description, if g*E = E, then g leaves the smooth 
quadric W{E) fixed and does not exchange the two rulings. Hence, g G SL{U) x 
SL{U'), for some complex vector spaces U and U' of dimension two, acting over 
p3 = (g) u') and one has to check how SL{U) x SL{U') acts on the space . 
Notice that the first SL{U') does not change anything so, for any E e Ml^^i^k) 

(3) SL{U') C Sym{E) 
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and it follows that 



(4) MIUk)/SLiy) = C/SLiU) 



(compare it with Corollary p. 61 ). This description was performed for = 2 by 
Hartshorne ( |Par78|| ) and in the general case by Spindler and Trautmann ([ [ST9CI|| ), 



although they considered U = U'. 

Let us now see that under this isomorphism, the isomorphism class of the bundle 
is uniquely determined by the S'L(2)-class of its associated Hankel matrix. Indeed 
we have 



Lemma 4.11. (i) For every g G SL{2) 

( 'T'' A ) -^<^--) 

with respect to the standard J. 
(a) For any {k + 1) x [k + 1)-Hankel matrix H 

{S^'^g)' ■ [4(xo,xi)|4(x2,X3) ■H]-(^^^ ^ S^g^ ^ [hix'o,x[)\Ik{x'^,x'^) 

being H' = [S'^gY ■ H ■ S'^g a Hankel matrix. 

(ill) The isomorphism MI^,{k)/ SL{V) = G^/SL{U) takes the SL{V) class of the 
bundle to the SL{2)-class of its associated Hankel matrix given in 

Proof. We will prove [Hi) since (z) and (ii) are easy to check. Given E a special k- 
instanton bundle choose coordinates such that {X0X3—X1X2 = 0} G W{E). Take two 
fc-jumping lines Li and L2 such that H^[E\i^) and H^{E\i^) are orthogonal spaces 
into W with respect to J (here H^{E\l.) are considered into W by the monad (|l|) ). 
Moreover we can assume that Li = {xq = Xi = 0} and L2 = {x2 = X3 = 0}. With 
this choice of coordinates there exists a matrix A representing E as in (0) containing 
a Hankel matrix H. SL(2) acts on these choices, the S'L(2)-class of H is uniquely 
determined and characterizes the S'L(V)-orbit of E. □ 

As a consequence we obtain the following nice description 
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Corollary 4.12. Let Ml^^ik) he the moduli space of special k-instanton bundles 
on ¥^ . Then all points of MI^'i{k) are semistable for the action of SL{V) and the 
only not stable ones have a section vanishing on a line with multiplicity (k + l) . All 
the not stable orbits contain in the closure the unique orbit of bundles having two 
distinct sections each one vanishing on a (different) line with multiplicity (A; + 1). 

Proof. It follows from the isomorphism Corollary |3.6| and Lemma [4.11| . □ 

Proposition 4.13. Let E be a k-instanton bundle such that SL{U') C Sym{E) 
having a fine action. Then E is special. 

Proof. By the assumption there is a C* C SL{U') with a two-dimensional eigenspace 
in V of positive weight. Then there is a matrix representing E such that in the 
first X (A; + 1) submatrix only the coordinates of this eigenspace appear. It fol- 
lows that the line spanned by this coordinates is a fc-jumping line. Since Zk{E) is 
5'L([/') -invariant and not empty, it is one-dimensional and the result follows from 
Corollary □ 



Let us now briefly describe different group actions of G C Sym{E), being E a 
special fc-instanton bundle. 

Let E he a special fc-instanton bundle represented by a matrix A (see Nota- 
tion In general we have SL{2) C Sym{E) acting in the following way. If 



a (3 
7 5 



e SL{2) 



then the action on W is determined by 



a- Id (3-H 
7-/7~i 5 -Id 



G Sp{W) 

while the action on / is trivial and the action on V is given by 



Xq 
Xi 
X2 



axQ + 7x2 
axi + 7x3 
(3xq + 5X2 
(3xi + 6x3. 



Let us now describe some particular case. Consider the Hankel matrix H given by 
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It corresponds to the form x^y^ which, for /c > 2, is the only form such that A 7^ 
and such that the stabihzer has dimension 4. In this case, C* • SL{2) C Sym{E) 
where SL{2) acts as above and C* acts in the following way. For t e C* we have 



t 



e SL{I), 



e Sp{W) 



and the action on V is determined by 



Xq 



tXn 



Xi I— > t~^Xi 
X2 ^ tX2 
X3 1-^ t~^X3. 

A matrix description of the instanton bundle E with Sym{E) — C* • SL{2) is 

/ Xq Xi X3 X2 \ 



(5) 



V 



Xq Xi X^ X2 



I 



Finally, let us mention that bundles E having two distinct sections each one 
vanishing on a (different) line with multiplicity + 1) are quite interesting, because 
they are characterized by the property Sym{E) ~ SL{U) ■ C* with a fine action. 

We will end this section given a more precise description of the correspondence, 
introduced in Section 2, between special /c-instanton bundles and linear systems g\_^-^ 
without base points. More precisely we will prove 
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Theorem 4.14. Let MI^3[k) be the moduli space of special k-instanton bundles on 
P^. There is a natural morphism 

which takes E to the branch locus defined by its pencil of sections and which factors 
through 

0' : Mr^,{k)/SL{y) — ^ p2V5L([/). 

0' is finite of degree equal to degiGiV^ ,T{S^~^^U))) = = Ck+i, where Ck+i is 

the {k + l)-th Catalan number (see e.g. j^UKZU^; pag. 239). 



The above theorem appears in |[New81|| for k = 2 and in this case Mips (2) / SL(y) 
is isomorphic to A^. 

' '^^^ concerning 



This resuh answers the question posed by R. Hartshorne in ||Har78b 



the relation between the cross-ratio of the four branch points of a g^ and the orbits 
for the action of SL{U). Indeed, Hartshorne asked if the cross-ratio determined 
uniquely the orbit and P. Newstead showed that for each value of the cross-ratio 
there are exactly two orbits in with one exception (See [New81| for more details). 



Proof of Theorem t4.14 



By the isomorphism (|), MI^sik)/SL{V) ^ G^/SL{U). Moreover, by Lemma pl 
this isomorphism takes the S'L(l^)-class of the bundle to the 5'L(2)-class of its as- 
sociated Hankel matrix given in (]^). Hence, we will prove that there exists a finite 
morphism 



: G^/SL{U) 



P^ySL{U) 



of degree equal to Ck+i- To this end, let G = Gr{F^ ,¥{S''^^U)) and denote by pij 
the Pliicker coordinates on G. Define 



R : 



G 



(po,i, ■ ■ ■ — ' (qi- 

-- J2f,+u=m, ^<^(^ - (compare it with 



■ q2k+i) 



where, for any 1 < m < 2k + 1, qm 
[ [Ira88|] ; Pag. 41-42). 
Claim: R is induced by the projection from the linear subspace defined by 



q2k+i = 
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which is disjoint from G. 

Proof of the Claim: First of all recall that, for any {^o^ "^i^ "^2; "^3} C {1, ■ ■ ■ , /c}, the 
Pliicker coordinates p^j verify the following Pliicker relations 

We will prove by induction on m = fj, + u, that if gi = ■ ■ ■ = q2k+i = then 
Pij = for any pair If m = 1, the assumption gi = implies po,i = 0. Let us 

assume p^^^i, = for /i < z/ such that ji + u = m and we will see that p^^^ = for 
any fi < u such that fi + u = m+ 1. Considering the Pliicker relations for a suitable 
sets of indices and by induction hypothesis we get 

Pi,jPk,i = 

for any i < j, k < I with i + j = k + l = m+ l. Hence, from the assumption 

= Qm+l = Yl ^)^/^''^ 

we deduce that p^^u = for any fj, < u with /x + z/ = m + 1, which proves what we 
want. 

It follows from the claim that i? is a finite morphism. Moreover, since G'' C G, 
by [[TraS^I ; Pag. 41, R induces a finite morphism 

iP : G^/SL{U) — > ¥^^/SL{U) 



of degree equal to Ck+i which, by Theorem |3]^, sends the S'L(f/)-class of the linear 
system g\j^^ associated to the 5'L(V^)-class of a bundle to the branch locus defined 
by its pencil of sections. □ 

Remark 4.15. A general special instanton bundle with C2 = 2 has symmetry group 
SL{2) ■ Gq where Gg is binary dihedral of order 8. There is a central extension 

^ Z2 ^ Gs ^ ^2 © Z2 ^ 0. 

In fact Z2 © Z2 = -D4. If the quartic form corresponding to (oq, . . . ,04) satisfies 
/ = then Sym{E) = SL{2) ■ G24 where (^24 is binary tetrahedral of order 24. 
There is a central extension 

^ Z2 G24 A ^ 0. 
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If the bundle has only one section vanishing on a line with multiplicity 4 then the 
quartic form has a double root and Sym{E) = SL{2) ■ Giq where Giq is binary 
dihedral of order 16. The bundle with two sections each one vanishing on a different 
line with multiplicity 4 has Sym{E) = SL{2) ■ C* and it is characterized by the 
property dim Sym{E) > 4. 



5 Link between Sym and Zk 

The goal of this section is to study the symmetry group Sym{E) of a fc-instanton 
bundle E such that ZkiE) ^ 0. 



Theorem 5.1. Let E be a k-instanton bundle on such that Zk{E) contains a 
double point. Then there exists C C Sym^{E), where C has a fine action. 

Proof. By assumption and ||Ski97|| there is monad representing E with 

/ Xq Xi \ 
A= •.. ■•. H{xo,Xi,X2,X3) 

\ Xq Xi J 

0} G Zk{E) with coordinates {poi, . . .^13,^23) = (0, 

i<j . 



and the hne {xq = xi 



corresponds to a double point. By Remark pl6| , Zk{E) is given by PijAiJA 
The crucial point is that 



0,1) 
= 0. 



A' 



( X\ 
\ 



Xq Xi 

represents another instanton E' . Moreover 



if(0,0,X2,X3) 



AoJA\ = A'oJA'l, AoJAi = A'oJA'l AiJA\ = A\JA 



tit 

2) 



AiJAl 



A'lJA'l 



A2JAI = A'2JA'l = 

and A'qJA'{ = while, in general, AoJA{ ^ 0. By assumption the variety X]j<j Pij^iJ^^j = 
contains a line tangent to the Pliicker quadric P01P23 — P02P13 + PosPu = in the 
point (0, . . . , 0, 1). Hence the system 

Po2AoJAl+posAoJAl+puAiJAl+priAiJA}^ = 
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has a nonzero solution (po2,Po3)Pi2,Pi3) which gives the hne (0, spo2, spos, spi2, spi^, t) 
where (s, t) G P^. Notice that this system is the same which gives solutions in the un- 
knowns {poi, . . .pi3,p23) for Zk{E') which has now the solutions (m, spo2, spos, spu, spi3, 
where {u,s,t) e P^. Therefore, Zk{E') is obtained by cutting the Pliicker quadric 



with a plane which, in particular, means that it is a conic. Hence, by Corollary 



E' is special and by the Lemma 2.17 there exists a nondegenerate (k + l) x (A; + 1)- 
Hankel matrix K such that Ik{x2, x^) ■ K = H{0, 0, X2, x^). Define the morphism 



C 
t 



Sp{W) 
St 



where St is represented by the matrix 



St 



Id 




tK 
Id 



We have 



A' -St 



Xq Xi 



Xq Xi 



\ 



H{0, 0, txo + X2-, txi + xs] 



J 



Hence 



A -St 



Xq Xi 



\ 



Xq Xi 



\ 



II{xo, Xi, tXo + X2-, tXi + X-i] 



J 



and C C Sy7iiP{E) as we wanted. 



□ 



Remark 5.2. With the notations of the above proof, there is a subgroup A iso- 
morphic to C* given by xq i— > txo, Xi t— > txi, X2 ^ t~^X2, X3 ^-^ t'^x^ such that. 



by Corollary O, lim(_^o X{t) ■ E is still a fc-instanton bundle with four- dimensional 
symmetry group as in (H). 



Theorem 5.3. Let E be a k-instanton bundle on ¥^ such that Zk{E) contains two 
distinct points. Then there exists C* C SyivP{E), where C* has a fine action. 
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Proof. By assumption and 



Ski97|, there is monad representing E with 



A 



Xq X\ 



V 



H{xo,Xi,X2,X3] 



J 



and the hne Li = {xq = a;i = 0} G Zk{E). Since Zk{E) contains two distinct points, 
we can assume that L2 = {x2 = X3 = 0} G Zk{E). Moreover, from the fact that 
h^{E\L2) = k + 1 we deduce that there exists a nondegenerate [k + l) x (/c + l)-Hankel 
matrix M such that 



H{xo, xi, 0, 0) = Ik{xo, xi) ■ M. 



Defining 



S 



Id -M 
Id 



we get 



A - S = [4(a;o,a;i)| - M ■ Ik{xo,xi) + II{xo,xi,X2,X3)] = Ik{xo,xi)\H{x2,X3) 



where II{x2, x^) is a matrix of hnear forms only in X2,Xs- Therefore, there exists C* 
acting on U with weights —1, and on U' with weights 0, such that C* C Sym{E) 
having a fine action, which proves what we want. □ 
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